[MRE ML Seminar Series] Convex Optimization for Machine Learning

Abstract

This seminar explores the fundamental connections between convex optimization and machine learning, demonstrating why convex optimization serves as a crucial mathematical foundation for many ML algorithms. Unlike general optimization problems that are extremely difficult to solve, convex optimization problems such as least-squares, linear programming, and semidefinite programming can be solved reliably and efficiently with global optimality guarantees. The presentation establishes that many machine learning algorithms inherently depend on convex optimization, making it one of the few optimization classes that can be practically solved while providing insights into the intrinsic properties and structures of ML algorithms.

The seminar examines machine learning from four distinct perspectives: statistical (covering Maximum Likelihood Estimation and its equivalence to minimizing KL divergence and mean-square error), computer science (focusing on neural network architectures and hyperparameter optimization), numerical algorithms (including stochastic gradient descent and backpropagation), and hardware acceleration (utilizing GPU parallelism for performance enhancement). Through practical examples ranging from linear regression and support vector machines to deep neural networks, the presentation illustrates how these perspectives converge to create effective learning systems, with particular attention to how convex formulations enable robust solutions even when constraints are added.

The presentation concludes by exploring deep learning architectures, specifically convolutional neural networks (CNNs) for image processing and recurrent neural networks (RNNs) for sequential data, while discussing their applications across diverse domains including medical diagnosis, autonomous vehicles, IoT systems, and cybersecurity. The seminar emphasizes the practical impact of these technologies in industry, highlighting major players like Amazon, Google, Facebook, and Nvidia who are leveraging AI for transformative applications, and underscores how the mathematical rigor of convex optimization continues to underpin the reliability and scalability of modern machine learning systems.